What is robots.txt?
What is robots.txt?
358
26-Nov-2020
Updated on 26-Nov-2020
Anshu Dwivedi
27-Nov-2020A robot .txt file tells search engine crawlers which pages or file the crawler can,t request from your site. This is used mainly to avoid overloading your site with the request, it is not a mechanism for keeping a web page out of google.